22 research outputs found
Zero-Shot On-the-Fly Event Schema Induction
What are the events involved in a pandemic outbreak? What steps should be
taken when planning a wedding? The answers to these questions can be found by
collecting many documents on the complex event of interest, extracting relevant
information, and analyzing it. We present a new approach in which large
language models are utilized to generate source documents that allow
predicting, given a high-level event definition, the specific events,
arguments, and relations between them to construct a schema that describes the
complex event in its entirety. Using our model, complete schemas on any topic
can be generated on-the-fly without any manual data collection, i.e., in a
zero-shot manner. Moreover, we develop efficient methods to extract pertinent
information from texts and demonstrate in a series of experiments that these
schemas are considered to be more complete than human-curated ones in the
majority of examined scenarios. Finally, we show that this framework is
comparable in performance with previous supervised schema induction methods
that rely on collecting real texts while being more general and flexible
without the need for a predefined ontology
Assessing the number of ancestral alternatively spliced exons in the human genome
BACKGROUND: It is estimated that between 35% and 74% of all human genes undergo alternative splicing. However, as a gene that undergoes alternative splicing can have between one and dozens of alternative exons, the number of alternatively spliced genes by itself is not informative enough. An additional parameter, which was not addressed so far, is therefore the number of human exons that undergo alternative splicing. We have previously described an accurate machine-learning method allowing the detection of conserved alternatively spliced exons without using ESTs, which relies on specific features of the exon and its genomic vicinity that distinguish alternatively spliced exons from constitutive ones. RESULTS: In this study we use the above-described approach to calculate that 7.2% (± 1.1%) of all human exons that are conserved in mouse are alternatively spliced in both species. CONCLUSION: This number is the first estimation for the extent of ancestral alternatively spliced exons in the human genome
The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics
With an increasing number of parameters and pre-training data, generative
large language models (LLMs) have shown remarkable capabilities to solve tasks
with minimal or no task-related examples. Notably, LLMs have been successfully
employed as evaluation metrics in text generation tasks. Within this context,
we introduce the Eval4NLP 2023 shared task that asks participants to explore
prompting and score extraction for machine translation (MT) and summarization
evaluation. Specifically, we propose a novel competition setting in which we
select a list of allowed LLMs and disallow fine-tuning to ensure a focus on
prompting. We present an overview of participants' approaches and evaluate them
on a new reference-free test set spanning three language pairs for MT and a
summarization dataset. Notably, despite the task's restrictions, the
best-performing systems achieve results on par with or even surpassing recent
reference-free metrics developed using larger models, including GEMBA and
Comet-Kiwi-XXL. Finally, as a separate track, we perform a small-scale human
evaluation of the plausibility of explanations given by the LLMs
Human-in-the-Loop Schema Induction
Schema induction builds a graph representation explaining how events unfold
in a scenario. Existing approaches have been based on information retrieval
(IR) and information extraction(IE), often with limited human curation. We
demonstrate a human-in-the-loop schema induction system powered by GPT-3. We
first describe the different modules of our system, including prompting to
generate schematic elements, manual edit of those elements, and conversion of
those into a schema graph. By qualitatively comparing our system to previous
ones, we show that our system not only transfers to new domains more easily
than previous approaches, but also reduces efforts of human curation thanks to
our interactive interface.Comment: 10 pages, ACL2023 demo trac